Image completion with large-scale free-form missing regions is one of the most challenging tasks for the computer vision community. While researchers pursue better solutions, drawbacks such as pattern unawareness, blurry textures, and structure distortion remain noticeable, and thus leave space for improvement. To overcome these challenges, we propose a new StyleGAN-based image completion network, Spectral Hint GAN (SH-GAN), inside which a carefully designed spectral processing module, Spectral Hint Unit, is introduced. We also propose two novel 2D spectral processing strategies, Heterogeneous Filtering and Gaussian Split that well-fit modern deep learning models and may further be extended to other tasks. From our inclusive experiments, we demonstrate that our model can reach FID scores of 3.4134 and 7.0277 on the benchmark datasets FFHQ and Places2, and therefore outperforms prior works and reaches a new state-of-the-art. We also prove the effectiveness of our design via ablation studies, from which one may notice that the aforementioned challenges, i.e. pattern unawareness, blurry textures, and structure distortion, can be noticeably resolved. Our code will be open-sourced at: https://github.com/SHI-Labs/SH-GAN.

视频效果旨在通过给定的输入视频序列预测每个帧的α哑光。在过去的几年中，深度卷积神经网络（CNN）的最新解决方案一直由深度卷积神经网络（CNN）主导，这已成为学术界和工业的事实上的标准。但是，它们具有内置的局部归纳性偏见，并且由于基于CNN的架构而不会捕获图像的全局特征。在处理多个帧的特征图时，考虑到计算成本，他们还缺乏远程时间建模。在本文中，我们提出了VMFormer：一种基于变压器的端对端方法，用于视频垫子。它可以通过视频输入序列从可学习的查询中对每个帧的α哑光进行预测。具体而言，它利用自我发挥的层来建立特征序列的全局集成，并在连续帧上使用短距离的时间建模。我们进一步应用查询来通过在所有查询上使用远程时间建模的变压器解码器中的交叉注意来学习全局表示形式。在预测阶段，查询和相应的特征图均用于对Alpha Matte的最终预测。实验表明，VMFormer在合成基准测试上的表现优于先前基于CNN的视频效果方法。据我们所知，这是第一个基于完整视觉变压器建立的端到端视频底漆解决方案，并对可学习的查询进行预测。该项目在https://chrisjuniorli.github.io/project/project/vmformer/上开源